Process and device for the parallel capture of visual information

ABSTRACT

In order to ascertain what a person really sees, a monitoring, picture recording system has at least one video camera ( 3 ) arranged in such a way that the captured images at least approximately match the field of vision observed by the person. For that purpose, at least one eye parameter, such as the viewing direction and/or the refractive power of at least one eye ( 2 ), is determined and the picture recording monitoring system ( 3 ) is controlled depending on the thus determined parameters. The position of the center of the pupil is determined by image sensors ( 4 ) associated to the eyes and the visual axis is derived therefrom. At least one video camera ( 3 ) secured in an adjustable manner to the head is oriented according to the visual axes. Focusing may also be adjusted depending on the refractive power of the eye. The parallel capture of visual information may be used in many fields, in particular for monitoring tasks and also for therapeutic uses.

BACKGROUND OF THE INVENTION

The invention concerns a process for collecting and recording visualinformation in parallel with an initial stereoscopic sensor systemrecording optical information, for instance the eyes of a person, and adevice for carrying out such a process.

In manual activities which must be carried out extremely accurately, theresult depends strongly on accurate observation during the activity.This is particularly the case for surgery on human or animal bodies. Ifthe person operating does not observe the tissue in the vicinity of theknife accurately enough, there is a danger that an important structuresuch as a nerve bundle will be severed.

In systems with high hazard potential, such as nuclear power plants,assembly and checking must be done very accurately with good visualobservation. Accurate observation is also essential in manipulations andtests during space flight, flying, and in complex terrestrial systems.

The desire to make what a person sees recordable is old, and until nowit has not been possible to accomplish it. The attempt to obtain imageinformation by sampling from the optic nerve does not take us to thatobjective because a significant portion of the signal processing andimage collection occurs only in the brain.

One or more video monitoring cameras recording the whole area that canbe examined by a person only show what is available, but not what theobserving person sees. In order to be able to analyze potentiallyincorrect action of the person, it is certainly convenient if monitoringcameras record all that is visible. In addition, though, it is necessaryto determine what is seen by the person observed.

A microscope headpiece which can be worn during surgery is known fromU.S. Pat. No. 4,395,731. The operating field is monitored by two videocamera placed in front of the eyes. The images from the video camerasare sent to monitors or image areas immediately in front of the eyes.Use of zoom optics makes it possible to magnify segments of theoperating field.

The eyes always remain directed to the image source (e. g., a CRT). Thatcan quickly lead to fatigue. It is also impossible to determine how theeyes comprehend the image provided to them. It could be that animportant image area in the image source is never realized as suchunless it falls on the portion of the retina that allows sharp vision.The focusing and alignment of the two cameras fixes the distancerequired between the cameras and the surgical field in order to get asharp view with proper perspective. If this distance changes because ofhead movements, the images shown on the monitors can be unsharp, and/ormay not be viewed by the eyes in such a way as to give athree-dimensional image. The ability for manual adjustment is hardlysuitable for matching the orientation of the cameras to short-termchanges in distance. Another disadvantage of the microscope headpiece isthe limitation of the field of view by the camera optics and by thelimited size of the monitors. Furthermore, changes in the field of viewthrough movement of the eyes on the image area of the cameras ormonitors are limited. An optical system which attempts to determine theobservations of the operating or controlling person in which the personsees only the images from the cameras severely limits the ability ofthis person to observe and does not show what areas of the imagesprovided are being looked at.

SUMMARY OF THE INVENTION

The objective of the invention is to determine what image information isactually seen by a person, or by any light-sensitive sensor system.

The solution according to the invention provides an image-recordingcontrol system, itself known from U.S. Pat. No. 5,341,181, acting inparallel with the eyes of the person or in parallel with alight-sensitive sensor system, the information record of which is madeto coincide at least approximately such that at least one opticalparameter of the eyes of or the image-recording initial system isdetermined and the corresponding optical parameter of the control systemis adjusted in dependence on it. However, as the known system from theU.S. Patent is limited to a single control system which does not operatestereoscopically, the optogeometric determination of the image seen isonly approximate. In contrast, according to the invention, an opticalsystem is used which determines the stereo base (generally theinterpupillary distance) and uses the directions of the fixation linesof the eyes to determine the fixation point. If the fixation point isknown, the parallel observation of the working or controlling person isaccomplished more accurately than in the system according to U.S. Pat.No. 5,341,181. Each new eye position is determined. The optical axes arepreferably determined in a coordinate system which is fixed in relationto the head and which therefore moves with the head. As the movements ofeach eye are produced by six external eye muscles, it is possible toprovide a sensor means which determines the stimuli to these muscles andderives from those the current eye position or the optical axes of theeyes.

Now the eye positions, or the directions of the visual axes, which areknown in a coordinate system linked to the head, are used to control thedirections of at least one, but preferably two cameras, through a mountconnected with the head. The at least one camera is then connected withthe head so that it can, like the eyes, carry out not only movements dueto head movements but also movements relative to the head. It isapparent that an adjustable optical imaging system, especially one withadjustable mirrors or the like, can be provided to adjust the solidangle region being imaged in place of camera movement. Potentialapplications arise from quite different areas. One such area is that ofvideo telephony, in which one of the speakers can observe the samething, to a certain extent, with the eyes of the user, without addedtechnical recording manipulations being required. In one preferredembodiment, cameras with CCDs are placed at the temple areas at bothsides of the head. Each camera is assigned to the nearer eye. A mountingsystem, in particular, a kind of gimbal mount which makes cameramovements possible is provided on a holder part which can be placed onthe head, and an actuating means is provided. A camera, or its opticalaxis, is moved, for instance, by the movement of a rod which runsparallel with the camera axis and is connected to the camera. The rod ismoved in a plane orthogonal to the rod by means of two linear drivesarranged essentially perpendicularly to the rod.

As the camera positions are somewhat displaced from the eye positions, ageometric correction must be taken into consideration for control of thecamera direction. This correction should assure that the two camera axesintersect at the same point as the intersection of the two visual axesof the eyes. No correction is needed for parallel visual axes.

In order to determine the fixation point of the eyes, it must be takeninto consideration that the line of fixation of the eye does notcoincide with the optical axis. The optical axis of the eye is the axisthat is essentially symmetrical in rotation with respect to the opticalstructure of the eye (cornea, iris, lens). But because the mostsensitive region of the retina, the Fovea centralis, is somewhat to theside of that axis, the line of sight or line of fixation is angledsomewhat with respect to the optical axis. The deviation between thesetwo lines can be considered as a standard difference, or it can bedetermined by fixing the eyes on a point of known position with aspecified head orientation, and then measuring the directions of thevisual axes of the eyes. From the known head direction, or the positionsof the eyes and of the specified point of fixation, fixation lines areconstructed so that the deviations between the visual axes and thecorresponding fixation lines can be determined.

On consideration of the different directions of the visual axes and theline of sight of an eye, it is possible to calculate the direction ofthe visual axes by means of the sensors from the known deviation in thedirection of the fixation line. Then the camera axes are aimed at thefixation point according to the directions of the line of fixation, sothat the point of fixation lies in the center of the image area. But ifthe cameras are aimed according to the visual axes, the fixation pointis somewhat to the side of the center of the image. The position of thepoint of fixation can than be marked or taken into consideration on thebasis of the known deviations in the presentation or evaluation of thetest images.

If the control observation system need not collect three-dimensionalimage information, an embodiment with only one camera is sufficient. Itis preferably placed in the forehead region between the two eyes. Thiscamera should also be aimed at the intersection of the visual axes or,if necessary, at the intersection of the lines of sight. In the absenceof an intersection, the direction should be parallel to the visual axes.

Instead of aiming at least one camera, it is possible to use a camerasolidly connected to the head, preferably directed forward, especially acamera with a wide-angle optical system. Then the fixation linederivable from measurement of the optical axis directions can be used todetermine the area of the image being looked at. By continuously markingthe current fixation point or by showing only an area of the imagearound the current fixation point, it is possible to determine what theperson being observed is looking at.

Processes and devices to determine the direction of the optical axes ofthe eyes are known from the field of photographic and video cameras.Automatic focusing on the image area being looked at follows from theaxial relations determined. To determine the axes, the eye isilluminated with infrared light, which is not perceived by the eye, andthe reflected infrared light, or the image of the eye, is imaged on animage sensor plane. The eye image is made up essentially of reflectionsfrom the cornea and the iris. As most of the infrared light which passesthrough the pupil is not reflected back, the image sensor and an imageevaluation unit connected to it can detect a difference in the boundaryregion between the pupil and the iris. If necessary, the shape of theiris, dependent on the direction of the eye, is also determined and thecenter of the pupil is established as the point of intersection of thetwo principal axes found in the image of the elliptical or perhapscircular outer margin of the iris. Then the visual axis is the linepassing through the center of the pupil and through the center of eyerotation, which must have been determined previously.

To determine the center of eye rotation, lines of fixation aredetermined successively for the eyes fixed on two known points offixation, with fixed known head position and direction. The points offixation are preferably two fiducial points solidly connected to thecamera mount and to the sensors for determining the pupil position, sothat the determination can be independent of the direction of the head.Then the center of eye rotation is essentially at the intersection ofthe fixation lines through the fixation points. As the center of eyerotation is on the optical axis of the eye, a correction must be appliedto increase the accuracy of the determination because of the deviationbetween the fixation line and the optical axis of the eye, as describedabove. The distance between the centers of rotation of the two eyes isthe same as the interpupillary distance. If both eyes are lookingessentially straight and horizontally to the front, toward infinity,both irises will be imaged as circles on both image sensor planes. Thedistance between the centers of the pupils is the interpupillarydistance, and the centers of the pupils each lie in a zero-deflectionposition. To orient the eyes parallel and straight to the front,independently of the head position, fiduciary points linked to themounting can, if necessary, be placed in front of each eye. Then theeyes are directed parallel and to the front if both the fiduciary pointsassigned to the two eyes coincide in the images of both eyes.

The size of the pupil is known by determining the boundary regionbetween pupil and iris. This pupil can if necessary also be used as ameasured parameter to control the aperture of the control system, thusvarying the light conditions like those in the eye. That is advantageousif one is more interested in the image actually perceived by the eyethan in a potentially better image recorded by the camera (e. g., inmedical applications). In order to have optimal light conditions forimage recording in the control system, the aperture is adjustedautomatically. Then the actual brightness and sharpness impression ofthe user is lost; this can be selectable if necessary.

In the process of determining the visual axes at the present state ofthe art, the infrared light reflected back by the eye is deflectedlaterally out of the beam path of the camera onto the image sensor bymeans of a semitransparent infrared mirror (dichroic mirror) which hasvery high reflectivity for infrared light and very low reflectivity forvisible light. That assures that the view of the image plane of thecamera is undisturbed. In the solution according to the invention, onthe other hand, the eye does not look either through an ocular or ontoan image-displaying surface, but directly onto the object. But in orderto determine the axes, at least one infrared mirror must be provided infront of the eye, which diverts at least the infrared image of the eyeonto an image sensor outside the field of view. If necessary, theinfrared illuminating light is directed onto the eye from at least onesource at the side through a mirror.

In the known applications of visual axis determination, the eye isalways near the ocular of the viewfinder, so that the eye imagedetermined by the infrared measurement is affected little if at all byreflections of ambient light. In the system according to the invention,the eye is exposed to the ambient light essentially without shielding.To prevent disturbance by ambient light of the image of the eye and itsevaluation, published German Patent Application 43 37 098 A provides amode of operation in which the image information based on the differencebetween phases with infrared illumination and ambient light and phaseswith only ambient light is evaluated. An extremely compact sensor foraxis determination is known from European Patent Application EP 602 895.

As, in the system of the invention, the eye is exposed to ambient lightessentially without limit, however, it is well illuminated and can ifnecessary allow determination of the pupil position with an image sensor(CCD) that is sensitive to visible light. Thus illumination withinfrared light can be omitted for the axis determination.

Along with the direction of the eye, its accommodation by adjustment ofthe refractive power of the lens is another important parameter of theobservation process. We can, in a first approximation, assume that therefractive power of both eyes is always adjusted to the fixation point.Correspondingly, the camera optical systems should be controlled so thatthe fixation point, the distance of which can be determined from thedistance between the eyes and the directions of the fixation lines(triangulation) is focused sharply. Determination of the distance of thefixation point by triangulation is sufficiently accurate, at least inthe near field. If necessary, the camera optical systems can also befocused sharply in the range of the fixation point by autofocusing.

Eye refractors are used to determine the refractive power of the eye.They measure essentially the spherical dioptric power, the astigmatism,and the astigmatism axis. As the astigmatism and its axis do not changeduring an observation process, at least as a first approximation, onlythe spherical dioptric power need be measured to control the refractivepower of the control system by means of measured parameters. Themeasurement of these parameters is made more difficult by the fact thatit must be done for arbitrary directions of the eye.

The change of the refractive power of the lens involves changes in theshape of the lens. Those can be determined by means of ultrasoundechography, at least in certain directions or sectioning planes. Tomeasure the eye along the optical axis, an ultrasonic signal is coupledinto the eye through a lead-in segment of water. The lens thickness isdetermined from differences in travel times of the echoes at the twoedges of the lens. Aside from the pure travel time measurements along anaxis, imaging ultrasound measurement processes are also used for theeye. They give cross-sections through the eye. In the system accordingto the invention the eye must have essentially unimpeded vision to thefront. Any coupling in of ultrasound that may be required must thereforebe done at the side of the eye. Then the form of the lens, from at leastone cross section, along with the current visual axis, must be used tocalculate the current refractive power.

Ultrasonic measurement has the advantage, compared to measurements withlight, that the interior of the eye, including the lens, can be examinednot only essentially along the optical axis but also in a direction inwhich the eye is transparent to light. However, the accuracy ofmeasurement is poorer and the conversions that are needed are moretime-consuming.

With the preferred optical measurement of the refractive power of theeye, it is important that the refractor be focused suitably for the eyebeing examined, and adjusted correctly for the distance between theeyes. It is also important to prevent the harmful effects of blinking onthe measurement. A refractor which, in an initial step, is adjustedoptimally for the eye being examined, which is directed at a fixationobject, is known from the published German patent application DE 29 37891 A. In another step, an adjustable test image structure, preferablylight beams of definite directions, is imaged on the retina by the lens.The light reflected back out of the eye from the image on the retina,and its change on adjustment of the test image, is analyzed to determinethe mean refractive power (spherical), the astigmatism (cylindrical) andthe axis of astigmatism (cylinder axis).

When the eye is movable, and its optical axis is known from ameasurement, it must be possible to image an infrared test image ontothe retina along the current visual axis. That can be done, for example,by producing an image with a light-emitting diode array, or with arear-illuminated liquid crystal display, which is deflected by anoptical system, focused if necessary, and directed through the pupilinto the eye. Imaging on the retina is done by deflection, again, andfocusing of an image detection system or infrared sensors. The testimages from various regions of the array, or from the display, can bedirected onto the eye with suitable optics along the current opticalaxes and focused if necessary. The imaging action of the optical systemof the eye can be determined by varying the test images. It is obviousthat the optical elements in the free field of view of the eye, andespecially the deflecting elements, or dichroic mirrors, theconstruction of which is known, for instance, from the. Japanese patentapplication JP 88865/1978, must be fully transparent for visible light.Not only flat mirrors but also curved mirrors may be used to deviate thebeam. It is possible, in essence, to illuminate the eye from anypossible direction of the visual axis from two opposite sides by meansof two quarter-spherical mirrors.

With the measuring system described above, having a transmitter matrixand a receiver matrix, sequential transmission and parallel receptioncan be provided in place of transmission of test images. In this way, atleast part of the light diodes in the diode array are turned on and offsuccessively, and the back-scattered images are used, preferably alongwith the axial position, also determined, to determine at least oneaccommodation parameter. In particular, the pupil position and the axisposition can be determined using a system with one transmitter and onereceiver matrix.

In place of a light-emitting diode array, it is also possible to use aninfrared diode laser which is adjustable in direction and, especially,in position. It must be possible to direct the laser beam, through anoptical system and a dichroic superimposing element, essentially alongthe current visual axis for imaging on the retina. It must be possibleto direct the back-scatter image from the retina, which is deflected outof the field of view and, if necessary, controlled by an adjustableoptical system, to at least one infrared sensor or, if necessary, to asensor array.

Continuous determination of the visual axes, the pupil diameter, and theaccommodation during a viewing process makes it possible to collectimage data in parallel with cameras and to perform a dynamic visionanalysis with this image material. Accommodation errors can be detectedby comparing the position of the fixation point with the measuredrefractive power or accommodation distance. By specifying fixationpoints which can be moved in three dimensions it is possible to examinehow the eyes follow these points. Dynamic vision analyses can alsodetect errors in eye guidance in reading text or music, and inparticular, in monitoring instrument displays. Aside from testing,control and diagnostic applications, therapeutic applications are alsopossible. Exercises in the boundary area of occurrence of recognizedvisual errors, especially accommodation limits or problems in exactaiming of both eyes at a single point (fusion problems) can be improvedwith interactive control of the visual activity. In music teaching, bothgood and poor reading of notes can be recorded and analyzed from therecords. Optimizing the visual process can also be important in sportssuch as tennis, shooting, and games with goal-keepers. For instance,persons in a control center could give instructions directly to a userby radio, e. g., to fix on certain objects.

View control of menu fields on a display screen is another area ofapplication of data collection according to the invention. Thispotential is particularly advantageous for operations in which theperson operating follows the operation on an image screen. The menufields are preferably labeled by symbols which are stored in the imageprocessing system for the purpose of recognition. The segments of thecontrol system image near the fixation point can be compared with thestored symbols using image processing. For this purpose, both the screeninformation and the image information from the control system areaccessible for image processing. Now if the controlled user eyes fixateon a menu field, or its symbol, for longer than a predetermined time, orinitiate the recognition process via a switch that can be actuatedmechanically or acoustically, the symbol being viewed is assigned, ifpossible, to a corresponding stored symbol. If the assignment issuccessful, other menu fields can be presented on the screen, ifnecessary, or a command is issued, after confirmation if necessary.

View control of menu fields through an ocular is known, for instance,from the field of video cameras. There only one visual axis need bedetermined to recognize the image area being looked at, or to make anassignment. If the head and eyes are freely movable, the zero pointposition and the orientation of the coordinate system fixed to the headmust be known, along with the visual axis, or the line of fixation, inthe same coordinate system in order to specify a fixation line inthree-dimensional space correctly and, if necessary, to be able tointersect with the image screen plane. Aside from determination of thevisual axis, the motions of the head must be monitored telemetrically.The cost of specifying a fixation line in three-dimensional space ishigh and, at least with small menu fields there is a danger that theproper field will not be recognized. Comparison of the symbols seen withpreviously specified symbols may be simpler and more efficient in mostcases.

Another application, basically similar to the view-controlled selectionof menu fields described above, is the recognition of instrumentdisplays being checked. In monitoring instruments, it is important thatindividual displays must be read at least once by the monitoring personin a specified time interval. The image material of the control systemmust, then, present this display at least once in the fixation regionwithin the specified time interval. That can be controlled by imageprocessing.

Another example application of the data collection according to theinvention with an image-recording control system is view-control of thecursor on a screen being viewed. Instead of assigning stored symbols toimage segments from the control system in the region of the fixationpoint by means of image processing, as described above, it is possibleto provide coordinate marks on the screen being viewed, which can bedetected in the image seen by means of image processing. The position ofthe fixation point or image center of the image of the screen recordedby the camera aimed according to the eye direction can then be convertedto screen coordinates. The screen coordinates of the fixation point ofthe eyes, and, therefore, of the camera, are transmitted to the screen,or to its control system, so that the fixation point can be represented,by means of its coordinates, on the screen being viewed, in the form ofa cursor.

View control of the cursor position on a screen being viewed has verymany potential applications. In operations in which the operating personfollows the operation on the screen, the view cursor can assist theorientation of persons assisting the operating person, making it easierfor them to understand the process of the operation, or makingexplanations easier and making it easier to follow instructions. Theview-controlled cursor can select menu fields, localize zoom regions, oreven be controlled by movements of the recording system generating theimage and, if necessary, by auxiliary devices.

It is obvious that the view control of the cursor can essentiallyreplace all known cursor controls, even the widely-used mouse. Thispotential replacement is particularly advantageous for persons withsevere physical disabilities. Even if only the eyes can be moved, viewcontrol of the cursor makes efficient and wide-ranging computer usepossible.

In the eye, the resolving power is greatest for the part of the imageformed on the Fovea centralis. Outside of this region, the resolutiondiminishes rapidly in all directions. It is actually vanishingly smallin the region of the blind spot. The image picked up by a cameranormally has essentially the same resolution over the entire image. Now,in order to produce the image information which the observer gets, theimage information from the control system can be reduced toward theedges by means of image processing, taking into consideration the actualresolving power of the eye. If necessary, one can even place asoft-focus lens, which gives the desired resolution, on the controlsystem camera. If the reduction in resolution is done only in theanalysis of the recorded image, by means of image processing, then it ispossible to compare the optimal observation with a real one, especiallyto determine what information is lost in the eye.

Such comparisons could be important in judging surgical malpracticecases, as there is generally negligence only in case there was anincorrect cut, the danger of which was detectable, or if an error wasseen by the operating person without him or her reacting to it. Anotherimaging monitoring system, which produces images with a perspectivedifferent from that of the operating person and without the limitationsof the human eye could provide image information not accessible to theoperating person.

It is obvious that other image-processing steps can also be provided,such as emphasizing contour lines or introduction of masks. In thisprocess, image material from other imaging sources can also be used toproduce images. Damping processes for the camera motions in case of veryrapid eye movements may be required, and filter processes may beprovided for the images produced.

It may be necessary to process the image material in a way similar tothe processing in the brain of the image recorded by the retina.Contrasts can be amplified, blurs sharpened, peripheral informationfilters, and movements of the head and the outside world compensated soas to produce a fixed image. Aside from the image material, the measuredparameters may also be filtered and/or processed to make possibleoptimal control of the control system and thus optimal image recording.

Similarly, image-altering processing steps can be used to gainunderstanding of other visual behavior or capabilities. The imagematerial from the control system is adapted, by image processing, toother vision, such as vision in case of retinal detachment, or thevision of animals, the eyes of which have other characteristics.

Aside from measurement of eye parameters and image recording by means ofa control system, it is also possible to provide for images beingdirected to at least one eye by means of a semitransparent opticaldeflecting element. The images added can be derived from the controlsystem and, if necessary in case one eye has a visual defect, show theimage that could be seen with optimal vision. Conversely, a person withnormal vision can be provided with the image material available to aperson with defective vision.

In surgical operations, it would often be desirable to be able toprovide the surgeon with images from other imaging processes, such asX-rays, ultrasonic images, computer tomograms and nuclear magneticresonance images, especially if they could be overlaid on the visiblesurgical field. If necessary, an electronically switched shutter elementcould be provided behind (as seen from the eye) the deflecting elementintroducing the images. Such a switching element would, for example,change from a transmissive to a reflective state due to liquid crystalchanges. In this way it is possible to superimpose the added images overthe actual background, or to view them without the background.

The control system can, if necessary, include a zoom optical system sothat, if required, a zoomed segment in the vicinity of the visualfixation point is observed. For example, superimposing the zoomed imagecan allow the operating person to switch from direct observation toobservation of an enlarged subfield in delicate phases of the operation(comparable to the known microscopic headpiece). The directions of theeyes while looking at the zoomed image is preferably no longer used tomove the optical axes of the cameras in the control system, but ratherfor menu-controlled selection of the parameters of the control system,rather as described in commonly owned U.S. patent application Ser. No.08/817,634 filed Apr. 24, 1997 and claiming priority of Swiss PatentApplication CH 3217/94-9 for surgical microscopes. If necessary,variations in the distance between the camera and the observed field arecompensated by autofocus, and especially by small changes in cameradirection.

Aside from picking up and, if necessary, superimposing in a zoom image,the control system can also collect images in a wavelength range outsidethe visible range, especially in the infrared range, or images with anenlarged field of view and/or with altered stereo base, and can, ifnecessary, provide them to the eyes.

The initial image-recording system is preferably at least one eye,especially a human or animal eye. Basically, though, all light-sensitivesensor systems, including, if necessary, even single sensors, systemswith multiple sensors, and image recording systems with animage-recording control system can be controlled. It is preferable thatat least one directional axis of the sensor system be included and thatthe at least one camera is aimed in dependence on that direction. Use ofan image-recording control system to control a light-sensitive initialsystem makes possible control of the situation-dependent behavior of theinitial system.

It is obvious that the image information from the control system and, ifnecessary, the parameters measured for the initial image-recordingsystem can be recorded.

Other device claims are conceivable, analogous to the process claims,which exhibit concrete features with which the corresponding processsteps can be carried out.

SUMMARY OF THE INVENTION

The drawings explain the invention by means of embodiments presentedschematically. These embodiments do not limit the invention.

FIG. 1. Front view of a head with two video cameras placed laterallyfrom the eyes and two sensors above the eyes to determine the visualaxes.

FIG. 2. Side view of a head with video camera, visual axis sensor, anddeflecting mirror.

FIG. 3. Side view of a head with video camera, visual axis sensor,deflecting mirror, image superimposing means, and switchable shutterelement.

FIG. 4. View of a system with an initial camera which can be aimedmanually, a control camera, and a means for measuring, controlling, andimage recording.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1 shows a head 1 with two video cameras 3 placed laterally from theeyes 2 and two measuring systems above the eyes 2, which includeeye-direction sensors 4, from the measurements or images from which itis possible to derive the positions of the centers of the pupils. Inorder to provide to the eyes 2 images from the aiming sensors 4 and, ifnecessary, light, especially infrared light from eye-direction sensors4, initial semitransparent deflecting elements 5 or mirrors, preferablydichroic mirrors, are placed in front of both eyes. If necessary themeasuring system also includes a sensor arrangement and a processingunit to determine the refractive index of each eye. The sensors 4, thesemitransparent mirrors 5 and the cameras 3 are fastened to a mount 6which can be placed upon the head 1.

It is preferable to provide gimbal mounts 7 as mounting devices for thecameras 3, with primary and secondary motion axes 7 a and 7 b which areessential perpendicular to each other. It is preferable, in order to aimthe cameras 3 according to the measured visual axis parameter in desireddirections relative to a coordinate system linked to the head, or to themount 6, to provide a two-dimensional adjusting system 8 for eachcamera. The adjusting systems 8 actuate rearward camera extensions bymeans of two linear drives, 8 a and 8 b, an initial drive and asecondary drive, operating along axes 7 a and 7 b. Here the initialdrive 8 a is fastened to the holder 6 and moves a primary rod 9, at theend region of which the secondary drive 8 b is fastened. The secondarydrive 8 b is connected to the camera extension through a secondary rod10.

FIG. 2 shows the head 1 with the holder 6 from the side, so that thecamera extension 11 can be seen. The side view also shows the mountingand orientation of the partially transparent mirror 5. The wires fromthe sensors 4, the adjusting systems 8 and the cameras 3 are preferablyconnected to a measuring, control and image-recording means. Thiscontrol means assures the orientation matching the eye direction and/orthe camera 3 focus corresponding to the measured refractive index of theeye. The control system also processes and/or stores the signals fromthe sensors 4 and the cameras 3.

FIG. 3 shows an embodiment in which, aside from measurement of thevisual parameters and image recording by the cameras 3 adjusted to matchthe parameters measured, it can also be provided that at least one eyecan be provided with images from an image source 13 fastened to theholder 6 through a second semitransparent optical deflecting element 12.In order both to superimpose the images on the background or to makeonly those images available to the eye, a shutter element is provided ifdesired, behind the deflecting element 12 as seen from the eye 2, whichcan be switched by an electronic unit 14. For instance, it changes froma transmissive to a reflective state due to liquid crystal changes. Thecontrol of the image source 13 and the unit 14 is provided from thecontrol system.

FIG. 4 shows an embodiment in which the initial sensor system recordingthe image information and also the control system each includes at leastone video camera. Instead of making the image viewed eye by oneviewable, the control system is intended to view images parallel to aninitial video camera. For this purpose, an initial video camera 16 ismoved manually by means of a control handle 17. To determine thedirection of the initial camera 16 it is provided with adirection-determining unit 18. The particular adjustment of the opticalsystem of the initial camera is, if desired, made determinable eitherfrom the sensor from the optical adjusting system of the camera 16 orfrom the measuring means 19 mounted on camera 16.

The parameters measured for the initial camera 16 are transmitted to acontrol system 20 which controls a second camera 21 (specifically itsaim and/or the adjustment of its optical system). The aiming is done byan aiming means 22 on which the second camera 21 is mounted. Ifnecessary the mounts 18 and 22 of the two cameras 16 and 21 areconnected together by a mounting system 23.

Applications in which the control system intentionally exhibitsdifferent parameters than the sensor system are also included within therange of the invention. For instance, the sensor system (e. g., videocameras in a robot) could work in a tele-zoom range, while the controlsystem exerts an oversight function in the wide angle range. Thus it canalso be advantageous that the control system works in a certain lightwave range parallel to the sensor system, which works in a differentlight wavelength range.

What is claimed is:
 1. A method for capturing image information parallelto the visual detection of image information by a pair of eyes, saidmethod comprising the steps of: a) providing an image-recording systemarranged in correspondence with said pair of eyes, and control means forchanging the alignment of said image recording system; b) measuring astereo base distance between said pair of eyes; c) measuring arespective alignment direction for each of said pair of eyes; d)establishing a fixation point of said pair of eyes based on said stereobase distance and said respective alignment directions; and e) aligningsaid image-recording system toward said fixation point to collect saidimage information.
 2. The method according to claim 1, wherein at leastone optical axis defined by at least one optical system is determined inmeasuring said respective alignment directions.
 3. The method accordingto claim 1, wherein said pair of eyes comprises the eyes of a person. 4.The method according to claim 3, wherein said image-recording system ismounted on the head of said person.
 5. The method according to claim 4,wherein said respective alignment directions are measured by a pair ofeye direction sensors located in front of said eyes.
 6. The methodaccording to claim 1, wherein said image-recording system includes acamera, and said camera is aligned toward said fixation point.
 7. Themethod according to claim 1, wherein said image-recording systemincludes a pair of cameras, and each of said pair of cameras is alignedtoward said fixation point.
 8. The method according to claim 3, whereinsaid image-recording system includes a pair of cameras, and each of saidpair of cameras is aligned toward said fixation point.
 9. The methodaccording to claim 5, wherein each of said pair of eye direction sensorsincludes a dichroic mirror in front of an eye and imaging means out ofthe field of view of said eye, said dichroic mirror passing visiblelight to said eye along said alignment direction of said eye andreflecting infra-red light between said eye and said imaging means toform an image of said eye at said imaging means.
 10. The methodaccording to claim 9, wherein said imaging means is a CCD, and aposition of the center of the pupil of said eye is determined in orderto determine an optical axis of said eye.
 11. The method according toclaim 9, further including an adjustment step of directing said eyes ata reference point linked to said head to determine said stereo basedistance.
 12. The method according to claim 10, wherein said opticalaxis is established through the center of rotation of said eye andthrough the center of the pupil of said eye.
 13. The method according toclaim 3, wherein said step of establishing said fixation point includesdetermining a line of sight for each of said eyes from said respectivealignment directions by means of an angular correction.
 14. The methodaccording to claim 13, wherein said angular correction is a standardangular correction for all persons.
 15. The method according to claim 3,wherein said angular correction is a non-standard angular correctiondetermined by fixing said eyes on a reference point of known positionwith a specified head orientation, measuring said respective alignmentdirection for at least one of said eyes, constructing said line of sightfor said at least one eye, and calculating the angular differencebetween said measured alignment direction and said line of sight. 16.The method according to claim 8, further including the step of measuringa refractive power of at least one of said eyes, said refractive powerincluding a spherical dioptric power of said at least one eye.
 17. Themethod according to claim 16, wherein said refractive power furtherincludes an astigmatism value and an astigmatism axis of said at leastone eye.
 18. The method according to claim 6, further including the stepof adjusting a focus of said camera to a distance between said cameraand said fixation point.
 19. The method according to claim 7, furtherincluding the step of adjusting a focus of each of said pair of camerasto a distance between said camera and said fixation point.
 20. Themethod according to claim 16, further including the step of adjusting afocus of each of said pair of cameras to a distance between said cameraand said fixation point, wherein said focus is adjusted based on saidmeasured refractive power.
 21. The method according to claim 17, furtherincluding the step of adjusting a focus of each of said pair of camerasto a distance between said camera and said fixation point, wherein saidfocus is adjusted based on said measured refractive power.
 22. Themethod according to claim 18, wherein said focus is adjustedautomatically.
 23. The method according to claim 19, wherein said focusis adjusted automatically.
 24. The method according to claim 1, whereinsaid captured image information is recorded.
 25. The method according toclaim 16, wherein said refractive power is recorded.
 26. The methodaccording to claim 3, further including the step of directing imagesfrom an image source to at least one of said eyes through asemitransparent optical deflecting element positioned in front of saideye.
 27. The method according to claim 26, wherein said image source issaid image recording system.
 28. The method according to claim 26,wherein said image source is an ultrasound image source.
 29. The methodaccording to claim 26, wherein said image source is an X-ray imagesource.
 30. The method according to claim 26, wherein said image sourceis a computerized tomography image source.
 31. The method according toclaim 26, wherein said image source is a positron emission tomographyimage source.
 32. An image recording system for capturing imageinformation parallel to the visual detection of image information by apair of eyes said image recording system comprising: at least one cameraarranged in correspondence with said pair of eyes, said at least onecamera having an optical parameter in common with said pair of eyes;control means for changing the alignment of said at least one camera andadjusting said optical parameter of said at least one camera; means forcontinuously establishing a fixation point of said pair of eyes andproviding signal information to said control means to align said atleast one camera with said fixation point; and means for continuouslydetermining said optical parameter of said pair of eyes and providingsignal information to said control means to adjust said opticalparameter of said at least one camera to be congruent with said opticalparameter of said pair of eyes.
 33. An image recording system accordingto claim 32, wherein said means for continuously establishing a fixationpoint includes at least one direction sensor for determining at leastone optical axis of said pair of eyes.
 34. An image recording systemaccording to claim 32, wherein said means for continuously determiningsaid optical parameter of said pair of eyes includes at least onerefractive power detector for determining a refractive power of saidpair of eyes, whereby the focus of said at least one camera is adjustedin accordance with said refractive power.